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was appointed and charged with development of institutional performance 
standards on critical success factors. The standards are listed in the 
document. A second task force, the Program Review Task Force, was charged 
with responding to Senate Bill 27. A third task force, the Performance 
Measures and Standards Task Force, was appointed to work on the special 
provision enhancing accountability. As part of a response to the special 
provision in Senate Bill 1366, the Performance Measures and Standards Task 
Force recommended sixteen measures for program accountability. The following 
measures and standards are included: (1) progress of basic skills students; 

(2) college- level courses; (6) program enrollment; (7) student satisfaction 
of program completers and non-completers; (8) goal completion of program 
completers; (9) curriculum student progress and success; (10) employer 
satisfaction with graduates; (11) employment status of graduates; and (12) 
client satisfaction with customized training. Each measure includes a 
description/definition, methodology and data source information, reporting 
periods and timelines, standards to be met, and a rationale for the standard. 
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Final Report on Senate Bill 1366 
Annual Review of Accountability Enhanced 
North Carolina Department of Community Colleges 



SBCC Approved 2/19/99 



The State Board of Community Colleges is responding to the special provision in Senate 
Bill 1366, Section 10.5: 

ANNUAL REVIEW ACCOUNTABILITY ENHANCED 

Section 10.5. The General Assembly finds that the current annual program review 
standards are not adequate to ensure that programs are meeting the needs of students, 
employers, and the general pubhc; therefore, the State Board of Community Colleges 
shall review the current standard to ensure a higher degree of program accountability 
and shall estabhsh appropriate levels of performance for each measure based on sound 
methodological practices. 

The State Board shall make an interim report to the Joint Legislative Education 
Oversight Committee and to the Fiscal Research Division on its improved 
accountability measures prior to November 1, 1998, and a final report prior to 
February 1, 1999. 

Background 

In its 1989 session, the North Carolina General Assembly adopted a provision (S.L.1989; 
C.752:S.80) that mandated that: 

The State Board of Community Colleges shall develop a “Critical Success Factors” 
hst to define statewide measures of accountability for all community colleges. Each 
college shall develop an institutional effectiveness plan, tailored to the specific mission 
of the college. This plan shall be consistent with the Southern Association of Colleges 
and Schools criteria and provide for collection of data as required by the “Critical 
Success Factors” list. 

The staff in the Department of Community Colleges collaborated with the Presidents’ 
Association, staff at the colleges, and the system’s planning committee to develop the list of critical 
success factors and measures in 1989-90. The State Board of Community Colleges adopted the 
critical success factors in July 1989. A matrix of factors and measures was approved, and the first 
report on the critical success factors was presented in April 1990. Annually since then reports are 

1 




3 



SBCC Approved 2/19/99 



presented to the State Board of Community Colleges. 

In 1993 the General Assembly passed a special provision on accountability, mandating the 
following: 

The State Board of Community Colleges shall establish standards for levels of 
institutional performance on those critical success factors that can be appropriately 
measured to indicate how individual colleges are performing in meeting the goals of 
the North Carolina Community College System. Each community college shall report 
its performance on these measures to the State Board. 

In addition, the legislation required that the State Board study models for measuring 
institutional effectiveness to use in reporting accountability information. Specifically, colleges were 
mandated to report on graduate placement rates and employer, graduate, and early leaver satisfaction 
with the college. 

To respond to the 1993 legislation, the Department of Community Colleges appointed the 
AccountabUity Task Force and charged it to develop institutional performance standards on critical 
success factors. 

Standards adopted are as follows: 

A. Where performance standards have been established by an outside accrediting/ 
hcensing agency, such standards were adopted by the system. Nursing requires a 75% 
passing rate for both RN and Licensed Practical Nursing. Insurance has a standard 
of 65%, and real estate requires a passing rate of no less than 15% below the state 
average. 

B. Where performance standards do not exist, the standard is designated as being 15% 
below the state average on any given measure. 

C. In reporting performance on colleges and setting standards based on the system 
average, a three-year system average measure is used. Each college’s three year 
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average performance is compared with this standard. 

While the Accountability Task Force was addressing the 1993 legislation, the Program 
Review Task Force was charged with responding to Senate Bill 27: 

1 . “Identifying any unproductive, low-quality, unnecessary, or duplicative programs . . . 
(Senate Bill 27, p. 86, Sec. 1 19 (a) (1) (d).) 

2. “Establish standards for the periodic review of community college programs including 
standards for the termination of programs.” (Senate Bill 27, p. 87, Sec. 1 19 (a) (6).) 

The accountability and program review task forces coordinated their efforts, resulting in a 
reco mm endation that each community college collect specified core measures for each academic 
program and submit to the department each fall. 

The measures and standards for the annual review of programs are provided below: 

A. Associate of Applied Science, Diploma and Certificate Programs 

1 . Enrollment by discipline (average of 1 0 students, vmduplicated headcount, for previous three 
years) 

2. Student goal accomplishment 

a. Program completion 

b. Other 

3. Student satisfaction with program 

a. Program completers 

b. Early leavers 

4. Employment rate 

a. Program completers 

b. Early leavers 

5. Employer satisfaction 

6. Certification/licensure passing rate, if applicable (Standard as set by outside certification/ 
Ucensure agency if such standards exist. All others are set by the Department of Community 
Colleges based on the previous three-year average— no less than 15% below the system 
average.) 

Standards are based upon a three-year average. The system average minus 1 5% of the system 
average is the rniiiimum standard for all measures except for enrollment and certification/licensure 
examinations that have agency standards. Failure to meet certification/licensure standards leads to 
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program termination or level 2 review (Documentation of Program Improvement and Continuation). 
If a program fails to meet more than three of the other eight measures (II. A. 1-5), then the program 
is terminated or moves into level 2 review. 

B. Associate of Arts, Associate of Science, and Associate of Fine Arts Programs 
1 . GPA of students identified as college transfer after two semesters in a four-year institution 
If a college transfer program fails to meet the state standard of 15% below the state average for two 
of three years, then the program is terminated or moves to level 2 review. 

The core program measures correspond closely to the measures of the critical success factors: 
program completion; student satisfaction with the program; employment rate; employer satisfaction; 
certification/Ucensure passing rate; and transfer student performance. The Annual Program Review 
was implemented the fall of 1995 as a trial data collection. Based on the 1996 data collection, 
system-level data on employer satisfaction with community college curriculum program completers 
were included in last year’s critical success factors report. 

In 1997 Dr. Barry Russell, Executive Vice President for the Department of Community 
Colleges, appointed the Accountability Monitoring Committee, which he chairs and which consists 
of departmental staff. The committee used the data on passing rates on licensure and certification 
examinations to identify programs at the system level that were not performing at an acceptable level. 
Program staff worked with college staff to identify “best practices” at the colleges where 
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passing rates exceeded expectations. Best practices are shared with college staff for the purpose of 
improving the programs at all campuses. 

Using the critical success factors standards, the Accountability Monitoring Committee 
identified programs at the 58 community colleges that failed to meet the standards. Dr. Elizabeth 
Johns, Vice President of Academic and Student Services, notified the presidents of these colleges of 
the programs that did not meet the standards and has asked for a response— either program 
termination or an action plan and timetable for improving the programs. Presidents responded with 
detailed plans for improvement, and staff are reviewing and monitoring the action plans. It is the 
intent of the department to work with colleges to improve programs. 

Study of Performance Measures and Standards 

To respond to the special provision enhancing accountability. President Martin Lancaster 
appointed in August the Performance Measures and Standards Task Force to study and to 
recommend accountabiUty measures and standards. Members of the task force are Dr. Willard Lewis, 
President of Isothermal Community College and chair; Dr. Steven Scott, President of Southeastern 
Community College; Dr. Cecil Groves, President of Southwestern Community College; Ms. Johnnie 
Simpson, Vice President for Instruction at Brunswick Community College; Mr. Jan Crawford, 
Director of Research and Planning at Fayetteville Technical Community College; Dr. Susan Allred, 
Vice President of Institutional Planning and Support Services at Forsyth Technical Community 
College; Dr. David Heatherly, Vice President of Instmction and Support Services at Coastal Carolina 
Community College; Dr. Dennis King, Vice President of Student Services at Asheville-Buncombe 
Technical Community College; Ms. Anne-Marie Knighton, member of the State Board of Community 
Colleges; Dr. Barry Russell, Executive Vice President of the Department of Community Colleges; 
Dr. Elizabeth Johns, Vice President of Academic and Student Services of the Department of 
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Community Colleges; Dr. Brenda Rogers, Vice President for Administration of the Department of 
Community Colleges; and Mr. Kennon Briggs, Vice President for Business and Finance of the 
Department of Community Colleges. Keith Brown, Associate Vice President of Planning and 
Research of the Department of Community Colleges, serves as staff to the task force. 

As the review of previous legislation and actions taken by the State Board of Community 
Colleges indicates, the Department of Community Colleges currently responds to two mandates: (1) 
the critical success factors and performance measures; and (2) the annual program review with 
associated performance standards for each program at each college. Because there is much overlap 
between the standards and the measures, the Performance Measures and Standards Task Force 
decided to review existing measures and to develop a single set of measures for evaluating 
effectiveness. This review and the final recommendations are consistent with the following two 
objectives in the 1999-2001 Strategic Plan adopted by the State Board of Community Colleges in 
June 1998: 

Objective 9.1 By 2000 review and revise the requirements of system-level accountability processes 
to reduce redundancy in reporting, improve data for decision making, and improve the 
cost effectiveness of the processes. 

Objective 9.2 By 2000 estabhsh system benchmarks of core indicators that measure programs and 
services. 

The Performance Measures and Standards Task Force held its first meeting on September 3 
at Guilford Technical Community College, at which time the purpose of the task force was reviewed 
and a tentative calendar was distributed. Keith Brown provided background information and 
distributed articles and other materials for the members to review before the next meeting. At 
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the second meeting, held on October 7, the task force reviewed performance measures, categorized 
each measure as to its strength, and evaluated the availability of the data to support the measure. 

Based upon its evaluation of measures of program and institutional effectiveness that were 
identified in the Uterature and that are currently used by our colleges, the Performance Measures and 
Standards Task Force recommended sixteen measures for program accountability. Staff drafted 
information about each of the sixteen measures: description/definition; data collection methods, 
analysis of data, and source of the data; and the reporting periods and timelines for collecting the 
data. Staff disseminated the draft of the sixteen measures and the interim report prepared for the 
Joint Education Oversight Committee by posting it on the Department of Community College’s Web 
page. 

Dr. Peter Ewell, Senior Associate at the National Center of Higher Education Management 
Systems (NCHEMS), is serving as a consultant in the study of performance measures and standards. 
Dr. Ewell reviewed the proposed measures and provided a written critique with recommendations 
for standard setting. At its December 1 5 meeting, the task force reviewed the measures and discussed 
with Dr. EweU by a conference call his critique and recommendations. As a result of this discussion, 
the performance measures were reduced to twelve and the method of setting the standard for each 
measure was defined. Dr. Ewell agreed to provide further assistance in developing the standards, 
using existing performance data provided by the Department of Community Colleges and data from 
other community college systems when available and when appropriate. 
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Recommendations 

The task force recommends the following: 

A. The proposed set of measures and standards for each measure will replace the existing aimual 
program review and critical success factors standards. 

B. Standards will be applied at the institutional level, except for licensure/certification data for 
which each exam at each college must meet the standard. 

C. To ensure sound methodological practices as required in the special provision, the 
Department of Community Colleges will review data collection procedures and analysis for 
reliability and validity. 

D. Periodic evaluations of the standards will be conducted by the Department of Community 
Colleges and, based upon the evaluations, modifications to the standards will be 
recommended to the State Board of Community Colleges. 

E. The following twelve performance measures will replace the Critical Success Factors and the 
Annual Program Review accountability measures: 

1. Progress of basic skills students 

2. Performance of college transfer students 

3. Passing rates for licensure and certification examinations 

4. Passing rates of students in developmental courses 

5. Success rate of developmental students in subsequent college-level courses 

6. Program enrollment 

7. Student satisfaction of program completers and non-completers 

8. Goal completion of program completers 

9. Curriculum student progress and success 

10. Employer satisfaction with graduates 

1 1 . Employment status of graduates 

12. Client satisfaction with customized training 
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Discussion of Measures and Standards 
1. Progress of Basic Skills students 

Description/definition: Basic skills students include all adult literacy students. The percentage of 
students who progress is based on three measures: (1) progressing within level, (2) completing the 
level entered or a predetermined goal, and (3) completing the level entered and advancing to a higher 
level. In the case of AHS (Adult High School) and GED (General Educational Development), 
students progress to a higher level when they move to a curriculum or occupational extension 
program. 

Methodology and data source: The indicator measures the progress of basic skills students through 
the basic skills program. A second indicator of the progress of basic skills students is an analysis of 
the number of students with an Adult High School Diploma (AHSD) or a GED who enter a 
curriculum or occupational extension program at the college. This indicator is a measure of success 
for the student in gaining additional training and for the System and colleges in providing a continuum 
of programs. 

All of the data on literacy students are entered at the college level. Data on the progression 
of students through the basic skills programs are collected and analyzed using the Literacy Education 
Information System (LEIS) at the Department of Community Colleges. To determine the number 
of students with an AHSD or GED enrolled in the System, analyses of the annual curriculum 
registration and extension registration data tapes are conducted. 

Reporting periods/timelines: Data are requested annually on students enrolled in Basic Skills 
programs in a community college between the beginning of the summer term and the end of spring 
semester. Colleges report the status of these students by August 15. The federal report completed 
by the department is due to Department of Education - Division of Adult Education and Literacy on 
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October 1. 

Standard: A fixed standard of 75% 

Rationale: The standard is based on data from the last five years for the system. For the last two 
years, the system average was 78%. However, the average over the last five years is 74%. 

2. Performance of college transfer students 

Description/definition: College transfer programs provide educational experiences that will enable 
transfer students to make the transition to a baccalaureate program and perform as well as the 
students who enroU as first-time fi;eshmen at universities. The purpose of this measure is to compare 
the performance of community college associate degree students who transfer to public North 
Carolina universities with students native to the four-year institution. 

Methodology and data source: The cohorts consist of college transfer students entering the public 
universities each fall. One cohort analysis compares the cumulative GPA of college transfer degree 
recipients at the end of two semesters to the cumulative GPA of native juniors. The other cohort 
analysis compares the cumulative GPA of college transfer students completing 24 hours at a 
community college to the cumulative GPA of native sophomores. The two cohorts will be combined 
for this analysis. There must be at least ten students for a community college to have this measure 
reported for accountability purposes. 

Performance data on students who transfer to a four-year public institution are provided by 
the University of North Carolina General Administration and include only those students who 
transferred to one of the 1 6 constituent institutions of the UNC System. No data are available from 
the private colleges and universities in North Carolina. 

Reporting periods/timelines: The UNC System provides data annually, between June and 

September. The data are published annually in the Critical Success Factors Report. 

10 
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Standard: 84% will attain an overall GPA of 2.0 or higher after completing one academic year at 
the public university. 

Rationale: The most recent data for associate degree recipients from community colleges report 
79% achieving a cumulative GPA of 2.0 or higher as compared to 89% for students at the junior level 
who originally enrolled in the public university. Data from the cohort completing 24 hours at a 
community college is not yet available; however, staff predict that the percentage of this cohort with 
GPAs of 2.0 or higher will be over 79%. Data from other states show that community college 
transfers consistently perform slightly lower than the native population, so this goal stretches the 
community colleges while also appearing to be reasonable given the national trends. 

3. Passing rates for licensure and certiflcation 

Description/definition: The percentage of first-time test-takers from community college graduates 
passing an examination required for North Carolina licensure or certification prior to practicing the 
profession. 

A licensure requirement for an occupation is one that is required by state statute for an 
individual to work in that occupation. Certification is generally voluntary but may be required by 
employers or an outside accrediting agency. Purely voluntary examinations will not be reported. 
Methodology and data source: The examination pass rates for each college are reported on a 
program-by-program basis. The pass rate for a particular college program will be calculated by 
dividing the number of first-time test-takers passing the examination by the number of first-time test- 
takers sitting for the exa min ation. 

Data are collected by the department from agencies issuing the license or certification. The 
data are collected and analyzed during the fall and reported in the spring. 

Reporting periods/timelines: Participating boards and agencies report data to the department 
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annually and colleges review the data before publication in the Critical Success Factors Report. 
Issues: A fixed standard, such as 70% or 80% passing rate, could have the unintended consequence 
of colleges eliminating programs that provide essential services to the community. For example, the 
emergency medical basic exam has had average System passing rates in the 60% to 65% range for 
the past three years. Even if only half of the students taking the test pass, this number of persons 
certified may be providing a needed service to communities. 

On the other hand, programs that have licensing/certification requirements should prepare 
students for passing the examinations. Poor programs with low passing rates should not continue to 
exist. Procedures need to be implemented to ensure that only students who have the potential to 
succeed are enrolling in these programs. To do otherwise is to mislead students, wasting their time 
and wasting taxpayers money. 

A fixed standard for each exa min ation fails to take into account the normal distribution of 
scores and average passing rates. The 70% passing rate will, in some cases, exceed the 
hcensure/certification agency’s standard required for institutions in order for their students to sit for 
the exa min ation (for example, insurance and real estate). In another case, the passing rates for the 
opticianry examination are lower for test-takers who have not completed the program at the 
community college than for community college graduates, and the passing rate for community college 
graduates of the opticianry program is well below the 70% standard (38% for the 1996-97 year). 
According to the 1 998 Critical Success Factors report, the examinations that had average passing 
rates for the system below 70% were Emergency Medical Technician— Basic (63%), Opticianry 
(38%), Life and Health Insurance (69%), Property and Liability Insurance (60%), Real Estate— Sales 
(61%), and Real Estate— Broker (65%). 

Standard: An aggregate institutional passing rate of 80% for all first-time takers of 
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licensure/certification examinations, plus no passing rate falling below 70% for any single 
examination. 

Rationale: The 1996-97 data report six examinations for which the system average was below 70%. 
Since passing licensure/certification examinations is an important step in becoming qualified for 
employment in the field, then high passing rates are expected. 

4. Passing rates of students in developmental courses 

Description/definition: The developmental course passing rates for all courses coded as 

developmental will be computed for each college. 

Methodology and data source: The department has developed a computer program to identify 
developmental courses, identify students who are enrolled in these courses, and calculate passing rates 
for these courses. An overall passing rate for all developmental courses at each colleges will be 
computed. 

Reporting periods/timelines: Annually data from each college will be sent electronically to the 
department and reported in the Critical Success Factors Report. 

Issue: The measures may encourage grade inflation in developmental courses. However, if this 
occurs, the next measure should show low performance on the measure of student success in 
subsequent credit courses. 
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Some colleges may have many developmental students enrolling with very low academic 
skills, whereas others may have students who simply need a refresher course. These differences may 
need to be taken into account. One approach is to use initial placement test scores as a control 
variable. Another approach is to use pre- and post-test scores to analyze gains. 

Standard: 70% passing rate for all developmental courses 

Rationale: In the absence of system-level data, the task force estabhshed 70% as an arbitrary starting 
point. Dr. Ewell had suggested 75% based on his knowledge of data from other states. However, 
members of the task force argued for a lower standard based upon their knowledge of grade 
distributions and withdrawals from developmental courses at their colleges. 

5. Success rate of developmental students in subsequent college-level courses 
Description/definition: The proportion of community college students who pass at least one 
developmental course and successfully complete at least 15 credit hours of subsequent college-level 
courses and obtain a cumulative GPA of 2.0 or higher. 

The purpose is to provide evidence that developmental courses equip students with the skills 
and knowledge necessary for success in their college studies. Once students have successfully 
completed the developmental courses, they should be able to pass curriculum courses. 
Methodology and data source: A computer program has been developed and is being implemented 
at the colleges that will identify developmental courses and identify students who are enrolled in these 
courses. This program will have to be expanded to track cohorts of developmental students and 
compare cumulative GPAs with non-developmental students. 

Reporting periods/timelines: Annually data from each college will be sent electronically to the 
department and reported in the Critical Success Factors Report. 

Issues: The high standard may encourage imposing entry requirements for certain curricula and have 
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the effect of reducing access for students. 

Standard: No statistically significant difference in the proportion of developmental students as 
compared to non-developmental students who obtain cumulative GPAs of 2.0 or higher. 
Rationale: A cumulative GPA of 2.0 is necessary to graduate from a program and to transfer to 
another college or university. The expectation is that developmental students who complete 
successfully the prescribed sequence of courses will be as successful as non-developmental students 
in their programs of study. 

6. Program enrollment 

Description/definition: The annual unduplicated headcount enrollment in a curriculum program. 
Methodology and data source: This indicator would measure the number of individuals enrolled in 
a given curriculum program, with enrollment meaning the student was enrolled as of the census date. 
The data are available from the colleges’ Curriculum Registration File. 

Reporting periods/timelines: Curriculum registration data are submitted to the department by the 
colleges at the end of each semester. In addition, the annual data are available from the Annual 
Program Review file submitted by each college in October. 

Standard: The current fixed standard of an average of 10 students over a three-year period is 
recommended. However, this measure is not recommended for use in performance funding 
since it is a minimum standard for program viability. 

Rationale: The current standard was acceptable as a mini m um for program viability. 
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7. Student satisfaction of program completers and non-completers 
Description/definition: This indicator reports the proportion of a sample of graduates and early- 
leavers who indicate that the quality of the college programs and services met or exceeded their 
expectations. 

Methodology and data source: These data are being collected annually as part of the Annual 
Program Review (APR) process. The data are collected by survey, with each college using a standard 
set of questions. A response rate of 50% for completers is suggested, and a minimum of 15 
respondents (total) will be required to report the data at the institutional level. No minimum response 
rate is proposed for non-completers. 

Reporting periods/timelines: The data are reported to the department each October as part of the 
APR. 

Standard: 85% of the combined respondents will report being satisfied with quality of college’s 
programs and services. 

Rationale: Historical data show that 90% of completers and 85% of non-completers are satisfied. 
The 85% standard is supported by data from South Carolina, Hawaii, Massachusetts, and Kentucky 
community colleges. 

8. Goal completion of program completers 

Description/definition: The proportion of graduates of certificate, diploma, and degree programs 
who report that their primary goal in attending has been met. 

Methodology and data source: These data are being collected annually as part of the Annual 
Program Review (APR) process. The data are collected by survey, with each college using a 
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Standard set of questions. A response rate of 50% is suggested, and a minimum of 1 5 respondents 
will be required to report the data at the institutional level. 

Reporting periods/timelines: The data are reported to the department each October as part of the 
APR. 



Standard: 90% of program completers will report goal completion 
Rationale: Historical data show that the system average is over 90%. 

9. Curriculum student progress and success 

Description/definition: To appropriately measure student success, a composite of three measures 
will be used: 

1 . Number completing a curriculum program with a certificate, diploma, or degree 

2. Number who have not completed a program but who are continuing enrollment in either 
curriculum or occupational extension programs 

3. Of the remaining non-completers, the number responding to a survey that they have met 
their primary goal for attending college 

None of the three measures in isolation are sufficient to indicate student success; however, the three 
measures taken together do represent student success. This composite indicator will consist of three 
measures, each reported separately for each college. It wiU be based upon tracking an entering cohort 
of students. 

The sum of the three wiU be divided by the total curriculum students in the cohort to compute 
an indicator of curriculum student progress and success. 

Methodology and data source: Entering cohorts will be defined each fall based upon the new 
students admitted to curriculum programs. The cohort will be tracked each fall to determine those 
who have graduated and those who have continued to be enrolled. Those who have not graduated 
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and are not enrolled will be surveyed. The actual number of non-completers who respond that their 
educational goals were attained will be included as goal completers. The sum of the graduates, 
students retained in curriculum or occupational extension programs, and non-completers reporting 
goal accomplished will be the number of students who are progressing toward or have completed 
their goals. This number divided by the entering cohort is the percentage reported as students’ 
achieving or progressing toward their educational goal. 

Reporting periods/timelines: Initial cohorts will be identified each fall. Graduates of programs are 
obtained fiom the Curriculum Student Progress Information System which is submitted each fall by 
the colleges. Student enrollment files as of the census date will be used to determine the number from 
the cohort who are still enrolled in either a curriculum or occupational extension program. Students 
who have not graduated and are not enrolled will be surveyed in the spring to determine if their 
educational goals have been achieved. 

Standard: 80% of the defined cohort will graduate, be retained, or report goal completion. 
Rationale: In the absence of system-level data. Dr. Ewell based this recommendation on data from 
Tennessee, South Carolina, Missouri, and Hawaii. 

10. Employer satisfaction with graduates 

Description/definition: The percentage of a sample of businesses who employ community college 
students indicating that their expectations of graduate performance have been met. This measure is 
intended primarily to determine the satisfaction of organizations whose employees have been trained 
through a community college. 
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Methodology and data source: Since 1994-95, all colleges are required to review all curriculum 
programs annually using a State Board of Community Colleges adopted Annual Program Review 
(APR) model. One measure contained in the APR is employer satisfaction. A common set of survey 
questions is used by each college to assess employer satisfaction. A response rate of 50% is 
suggested, and a minimum of 1 5 respondents will be required to report the data at the institutional 
level. 

Reporting periods/timelines: The data are submitted as part of the Annual Program Review each 
October. 

Issues: Colleges are having low response rates from employers. Some employers have company 
pohcies of not responding to surveys that assess the performance of their employees. The methods 
for collecting the data differ across colleges. A minimum response rate of 50% may be difficult, if 
not impossible, to obtain. However, a minimum number of respondents (15) and standard questions 
must be required for the data to be used. Reducing the frequency of surveying may increase 
responsiveness. 

Standard: 85% of employers report being satisfied with preparation of graduates. 

Rationale: Based on one year’s data for the system, 91% of employers were satisfied. Because of 
the limited amount of data and based upon Dr. Ewell’s knowledge of single-institution studies, he 
recommended a conservative standard of 85%. 

11. Employment status of graduates 

Description/definition: The proportion of identified community college completers achieving a 
marketable skill who obtain employment or increase their earnings within one year of last attendance. 
Methodology and data sources: With the North Carolina Common Follow-up System (CFS), we 
are now able to accurately track students’ employment status after they leave the colleges. The 
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Common Follow-up System (CFS) is a cooperative venture of the participating state agencies under 
the auspices of the North Carolina State Occupational Information Coordinating Committee (NC 
SOICC). The CFS provides a highly efficient and cost effective method for collecting follow-up 
information for education, employment, and training program participants statewide. The NC SOICC 
decided that the CFS would be maintained by the Employment Security Commission (ESC). 

Each year the agencies involved in the CFS submit unit record data on participants to the 
ESC. Among the agencies included in this process are the public high schools, community colleges, 
and the four-year public universities. Each agency’s data is matched against the Unemployment 
Insurance (UI) files and the other participating agencies’ files. A database containing information on 
employment, employer, quarterly wages, receipt of unemployment benefits, and participation in other 
agencies’ programs is returned to each submitting agency. The database each agency receives is 
limited to the participants that the agency submits for the data match. This is to say, the database 
received by community colleges has information only on community college students. 

Once the CFS database is received, it is matched against the Curriculum Student Progress 
Information System (CSPIS) database and the following year curriculum student registration 
database. This matching is conducted to determine demographic characteristics of the participants, 
such as students’ completion status at the end of the academic year and whether they re-enroll the 
following year. Students who obtain an associate degree, certificate, or diploma in the year given and 
do not re-enroll in any of the colleges the following year are defined as “exit completers.” Those 
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who do not obtain an associate degree, certificate, or diploma in the year given and do not re-enroll 
in any of the colleges the following year are considered as “exit non-completers.” 

Students who have wages in any quarter during the year are considered employed. Those 
who are found both in registration records and UI records but have no quarterly wages during the 
year are considered unemployed. 

Reporting periods/timelines: The data will be collected, analyzed and reported aimually in the CSF 
Report. 

Issues: Employment opportimities differ greatly across the state. There must be some adjustment 
made for community colleges that serve counties with very high unemployment rates. 

The CFS provides objective data on employment and earnings; however, it should be noted 
that only individuals employed in North Carolina and employed in jobs subject to the federal 
unemployment insurance guidelines will be found in the data match. If an individual is employed 
outside the state or is self-employed, no match will be found. This is not to say that such individuals 
are not employed; it simply means they are not found in the UI records. 

Standard: 90% will be employed. The percentage will be corrected for the average annual 
unemployment rate in the service area of each college in the following way: compute the difference 
between the state’s average annual unemployment rate and that of the service area and divide by 2. 
This amount will be deducted fi'om (for colleges with unemployment rates higher than the state 
average) or added to (for colleges with unemployment rates lower than the state average) the 90%. 
Rationale: Historical data show employment rates near 90%. Since graduates of community 
colleges most often remain in their local communities and college programs are designed to meet local 
needs, the unemployment rates of the service area will impact employment rates. Therefore, an 
adjustment based on the difference between the state’s unemployment rate and the service area’s 
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unemployment rate is necessary. However, we expect completers of community college programs to 
have an advantage over job-seekers in general, so the adjustment for the unemployment rate is only 
half of the difference between state and local unemployment rates. 

12. Client satisfaction with customized training 

Description/definition: The proportion of a sample of businesses for whom the college has provided 
services through the New and Expanding Industry program indicating that their expectations have 
been met will be reported. This indicator is intended primarily to measure the satisfaction of 
sponsoring organizations whose employees have been trained through customized programs provided 
by a community college under contract or agreement. 

The program examined by this measure is categorical and was created specifically to address 
employer needs. North Carolina's New and Expanding Industry training program provides the 
customized training that has been a major part of the state's economic development strategy. 
Methodology and data sources: Each college is requested to survey clients of customized training 
programs annually. Specific questions have been developed and each college is required to use these 
questions. 

Reporting periods/timelines: The data are collected annually by the Economic Development Section 
at the Department. 

Standard: 85% will report being satisfied with the New and Expanding Industry program. 
Rationale: Based upon general experience with client satisfaction surveys. Dr. Ewell recommends 
the 85% standard as a starting point. Clearly, this standard will need to be reviewed after actual data 
are available. 

In Table A the recommended measures are compared with the previous performance measures 
from the Critical Success Factors and the Armual Program Review. All existing accountability 

22 






SBCC Approved 2/19/99 



measures in the Critical Success Factors and Annual Program Review are recommended. The 
measures of success of developmental education programs are new measures. 



Table A. Current Use Status of Recommended Performance Measures in the 
North Carolina Community College System 



Measure 


Current Use 




Critical 

Success 

Factors 


Annual 

Program 

Review 


Perform. 

Program 

Budget. 


New Use 


Progress of Basic Skills Students 


X 


X 


X 




Performance of College Transfer Students 


X 


X 


X 




Pass Rates on Licensure/Certification 


X 


X 






Pass Rates in Developmental Courses 








X 


Success Rates of Students Taking 
Developmental Courses in Subsequent 
Related Courses 








X 


Program Enrollment 




X 






Student Satisfaction with College, 
Completers and Noncompleters 




X 


X 




Goal Completion of Program Completers 




X 


X 




Curriculum Student Progress and Success 
—Completion Rate 
—Continuing Enrollment 
—Goal Completion 


X 


X 


X 

X 


X 


Employer Satisfaction with Graduates 


X 


X 


X 




Employment Status with Graduates 


X 


X 


X 




Client Satisfaction with Customized 
Training 




X 


X 





Implementation 

A work group will be appointed to review the methodology for collecting and analyzing 
data to support the performance measures, with any changes in data collection and analysis 
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implemented in 1999-2000. The performance measures and standards will be approved by the 
State Board of Community College by July 1, 1999, and data collection and analysis will begin in 
1 999-2000. For those measures for which the Department of Community Colleges has collection 
procedures in place, the standards will be applied to the 1 999-2000 data. 

The Accountability Monitoring Committee, chaired by Dr. Barry Russell, will be 
responsible for monitoring the data and notifying colleges when they do not meet the 
accountability standards. The Accountability Monitoring Committee will continue to monitor 
colleges that do not meet minimum standards and require action plans to be submitted when 
standards are not met. Included in the action plans must be goals for improved performance and 
the timeframe for reaching the standard. The goal for demonstrating progress toward the 
standard will be based on the deviation from the standard and reasonable expectation for 
performance improvements. 

The Department of Community Colleges will ask that the General Assembly to enact 
legislation to adopt the performance measures and standards in place of the 1993 special provision 
on accountability and the 1993 Senate Bill 27, Sec. 1 19(a) (6) on annual program review. 

The Department of Community Colleges will review the measures and standards on a 
regular basis and propose modifications to ensure sound methodological practices. 
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